Temporally Coherent Bayesian Models for Entity Discovery in Videos by Tracklet Clustering

摘要

A video can be represented as a sequence of tracklets, each spanning 10-20frames, and associated with one entity (eg. a person). The task of \emph{EntityDiscovery} in videos can be naturally posed as tracklet clustering. We approachthis task by leveraging \emph{Temporal Coherence}(TC): the fundamental propertyof videos that each tracklet is likely to be associated with the same entity asits temporal neighbors. Our major contributions are the first Bayesiannonparametric models for TC at tracklet-level. We extend Chinese RestaurantProcess (CRP) to propose TC-CRP, and further to Temporally Coherent ChineseRestaurant Franchise (TC-CRF) to jointly model short temporal segments. On thetask of discovering persons in TV serial videos without meta-data like scripts,these methods show considerable improvement in cluster purity and personcoverage compared to state-of-the-art approaches to tracklet clustering. Werepresent entities with mixture components, and tracklets with vectors of verygeneric features, which can work for any type of entity (not necessarilyperson). The proposed methods can perform online tracklet clustering onstreaming videos with little performance deterioration unlike existingapproaches, and can automatically reject tracklets resulting from falsedetections. Finally we discuss entity-driven video summarization- where sometemporal segments of the video are selected automatically based on thediscovered entities.